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GAO 

^^^^^^^^^Accountabinty * Integrity * ReHabllity _ 

United States General Accounting Office 
Washington, DC 20548 


August 30, 2001 

The Honorable Donald H, Rumsfeld 
The Secretaiy of Defense 

Dear Mr. Secretaiy: 

This report summarizes the results of our review of the Joint Service 
Electronic Combat Systems Tester program, hereafter referred to as “the 
tester.” The tester is intended to provide the Air Force and Navy with an 
improved flight-line test capability to test the readiness of electronic 
combat systems, such as radar warning receivers and radar jammers, on 
their aircraft.^ These systems are vital in protecting the aircraft from 
enemy air defenses (i.e., surface-to-air missiles and antiaircraft artillery) 
and enemy aircraft. Our objective was to determine the schedule, cost, and 
performance status of the tester program. In addition, because the tester’s 
usage has disclosed serious reliability problems with electronic combat 
systems on Air Force and Navy aircraft, we have included a discussion of 
such problems in this report. 


R6SllltS in Bri6f Although schedule slippage and cost growth have occurred in the tester 

program. Air Force and Navy use of the new tester indicates that 
performance goals are being met and a useftd capability is likely to be 
achieved. The tester performed very effectively in testing—^so well, that it 
revealed numerous previously imdisclosed faults in electronic combat 
systems on Atr Force F-15C and Navy F/A-18C aircraft. For instance, we 
found in testing late last year that 12 of 13 aircraft at Langley Air Force 
Base and all 10 aircraft at Oceana Naval Air Station had one or more 
previously undiagnosed electronic combat system faults, indicating that 
the reliability of these systems is much lower than the services had 
previously believed. Because the tester works so well at disclosing faults, 
the Air Force and Navy plan to expand its use to other electronic combat 
systems on other fighter aircraft, including the F-16 and F-14. 

Widespread use of the tester could have several implications. First, the 
services could find that the readiness of their aircraft is lower than 


These systems are also referred to as “electronic warfare systems.” Electronic warfare is 
part of electronic combat. 
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previously believed. Because the tester effectively disclosed unknown 
faults, the Air Force found that 41 of 44 F-15Cs tested were not fully 
mission capable. Second, the services could be faced Avith additional 
demands for logistical support and maintenance. For example, while we 
observed the new tester being used on F-15C aircraft at Eglin Air Force 
Base, technicians took electronic combat system parts from other aircraft 
and installed them on the aircraft being tested before the testing could be 
completed. Maintenance officials told us that becaiose spare parts were in 
limited supply, it weis common for aircraft being tested to use cannibalized 
parts from another aircraft in order to be repaired. Third, although the 
expanded use of the new tester to other aircraft could make existing 
logistics and maintenance problems even worse, pilots would know more 
about the readiness and reliability of their self-protection systems. The 
failure to address these problems would encourage pilots to rely more on 
support from specialized aircraft designed to suppress enemy air defenses, 
such as the Navy/Marine Corps EA-6B. 

Because the new tester is so effective, we are recommending that you 
direct the Air Force and the Navy to consider expanding the use of the 
new tester beyond their fighter aircraft to other types of aircraft. The 
Department of Defense concurred with the findings and recommendation 
in our report. 


Background 


The armed services have a long-standing shortfall in their capability to 
adequately test electronic combat systems on aircraft and ships. From 
August 1989 through Jxaly 1991, we issued a series of reports identifying 
each service’s problems with their test equipment for electronic combat 
systems.^ To address these problems, in June 1993, the Air Force and Navy 
approved a Joint Mission Need Statement for a flight-line electronic 
combat systems tester to improve aircrafts’ electronic combat test 
capability. The Department of Defense designated the Air Force as the 
lead service, and the Air Force and Navy entered into a memorandum of 
agreement in December 1994 to establish a joint tester program. Following 


^ See Electronic Warfare: Reliable Equipment Needed to Test Air Force's Electronic 
Warfare Systems (GAO/NSIAD-89-137, Aug. 11,1989), Electronic Warfare: Faulty Test 
Equipment Impairs Navy Readiness (GAO/NSIAD-91-205, July 8,1991), Electronic 
Warfare: No Air Force Follow-Up on Test Equipment Inadequacies (GAO/NSIAD-91-207, 
July 17,1991), and Electronic Warfare: Faulty Test Equipment Impairs Readiness of 
Army Helicopters (GAO/NSIAD-92-128, Apr. 17,1992). 


Page 2 


GAO-01-843 Electronic Combat 



a concept development phase, an engineering and manufacturing 
development contract was awarded in March 1996. 

The tester has been developed to provide the Air Force and Navy with a 
flight-line test capability for aircraft electronic combat systems, to include 
both on-board systems and those moimted outside the aircraft in pods. 
The contractor for the tester, AAI Corporation, has developed a basic core 
test set that can be used with various aircraft. The basic core test set is 
supplemented by subsidiary test program sets and related software for 
each aircraft type and its specific systems. The tester provides an 
end-to-end test capability for electronic combat systems, including 
jammers, radar warning receivers, and other subsystems and their 
associated wiring. The tester inputs radio frequency signals into the 
aircraft’s antennae and then measures whether the signals were correctly 
received and the appropriate responses generated by the electronic 
combat systems. The tester can identify faulty wiring and also isolate the 
faulty system component to make the maintenance task easier. 

Developmental testing of the basic core test set and the test program set 
for the F-15C was completed in October 2000 and for the F/A-18C test 
program set in December 2000. Additional test program sets are to be 
developed for most of the current Air Force and Navy fighter aircraft 
equipped with electronic combat systems, and there will be growth 
potential for adapting the system for future aircraft. Quantities to be 
prociared include 56 Air Force and 40 Navy basic core test sets with test 
program sets for the F-15C and F/A-18C, respectively. The total planned 
procurement for the basic core test set is 121 for the Air Force and 188 for 
the Navy. Test program sets for other aircraft are to be subsequently 
developed and procured. 


Although Behind 
Schedule and Over 
Cost Estimate, New 
Tester Is Performing 
Effectively 


Schedule slippage and cost growth have occurred in the tester program. 
However, the Air Force’s and Navy’s use of the new tester indicates that 
performance goals are being met and that a useful capability is likely to be 
achieved. 

The development schedule for the new tester has slipped about 2 years 
from the original plan’s schedule because the difficully in designing the 
system was underestimated. 'This delayed the production decision for the 
tester imtil April 2001. Prior to the production decision, the services 
completed developmental testing but did not imdertake operational testing 
of the tester as planned. Operational testing was deferred because the lead 
test agency—the Air Force Operational Test and Evaluation Command— 
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was concerned that the tester contractor was still making design changes 
to the system and that operational testing should utilize articles that 
represent the final design to be produced. Consequently, additional 
developmental testing using available prototypes was substituted for 
operational testing to provide test data to support the production decision, 
ff operational testing of the tester’s final design identifies a need for 
further design changes, the testers procured would require retrofit. 

Regarding program cost, the cost under the initial development contract 
for the basic core test set and the F-15C and F/A-18C test program sets was 
originaliy estimated to be about $12 million. As of January 2001, the cost of 
the contract had increased to $28.9 million. Ultimately, the program’s total 
cost will be a function of future decisions regarding the extent to which 
other aircraft and electronic combat systems, such as the radar warning 
receivers and radar jammers on the Air Force’s F-16E and the Navy’s 
F/A-18E1/F, will use the new tester. These aircraft and their electronic 
combat systems will require the development and procurement of 
customized test program sets, as well as additional quantities of the basic 
core test set. 

According to the Air Force, the tester has performed effectively in testing. 
Developmental testing of the basic test set and the F-15C test program set 
was performed at Eglin Air Force Base firom March through October 2000. 
According to the Air Force’s developmental test organization, the tester 
met or exceeded expectations for all test objectives. For the key 
performance parameter of demonstrating at least 90-percent success in 
fault detection, the tester detected and isolated all faults. The testing 
disclosed that 29 of 31 F-15Cs actually had one or more faults in their 
electronic combat systems. 'The faults detected ranged from the 
identification of parts needing to be replaced inside the electronic combat 
systems (so-called Group B) to the wiring, antennae, and control units that 
connect the systems to the aircraft (so-called Group A). According to 
program officials, no existing tester has previously been able to test the 
Group A equipment as well as the Group B systems. Moreover, the new 
tester provides an ability to augment an electronic combat system’s 
internal system check (referred to as Built in Test, or BIT). In the past, if a 
system’s BIT indicated a fault, maintenance technicians were forced to 
remove the system components from an aircraft to retest them in the 
maintenance shop —a time-consuming and cumbersome process. The new 
tester provides a check against the BIT without the system’s removal from 
the aircraft. 
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Potential Implications 
From Widespread Use 
of New Tester 


Readiness Issues 


The Air Force used the tester to test operational 33rd Fighter Wing F-16C 
aircraft at Eglin about to be deployed to Operation Southern Watch in Iraq. 
After successful testing at the 33rd, it was then used to test F-16C aircraft 
at the 1st Fighter Wing at Langley Air Force Base. These aircraft are 
regularly deployed to Operation Northern Watch in Iraq. At Langley, 12 of 
13 F-16Cs thought to be fully mission capable actually had one or more 
faults in their electronic combat systems. The potential effects of some of 
these faults could have been that these aircraft would have entered 
combat with partially functioning protective systems; some of these faults 
would have left the systems nonfunctional. 

Navy test officials advised us that the tester also performed well with their 
F/A-18C aircraft, identifying faults that the Navy’s current test equipment 
had been rmable to identify. The Navy performed developmental testing of 
the basic test set and the F/A-18C test program set at Naval Air Warfare 
Center, Weapons Division, Point Mugu, California; Miramar Marine Corps 
Air Station, California; Lemoore Naval Air Station, California; and Oceana 
Naval Air Station, Vii-ginia, from September 1999 through January 2001. 

The Navy tested 16 aircraft in California, 14 of which had faults identified 
by the tester. Subsequently, 10 F/A-18C aircraft were tested at Oceana, and 
all were found to have unknown faults in their electronic combat systems. 
Each of the 10 aircraft had at least 3 faults disclosed by the new tester, and 
1 aircraft had 12 faults. 

Because the tester works so well at disclosing faults, the services plan to 
expand its use to other electronic combat systems on other fighter aircraft. 
The Air Force intends to use the tester also on its F-16s and the Navy, on 
its F-14s. 


Because the tester has a much greater ability to identify electronic combat 
system problems, it can identify faults that the currently used test 
equipment is not able to find. The disclosure of these problems could have 
significant implications for readiness levels, logistics, and maintenance. 
Additionally, the failure to address problems with the electronic combat 
systems could encourage pilots to rely less on their electronic combat 
systems and more on other specialized aircraft designed to suppress 
enemy air defenses, such as the EA-6B. 


The test results for the F-15C and F/A-18C have implications for readiness 
levels not only for those types of aircraft, but also for other aircraft using 
either the same or similar electronic combat systems (such as the F-15E 
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Logistics Issues 


and F/A-18E.) Readiness levels are lower than the services previously 
believed, since the F-15C and F/A-18C aircraft, which were previously (and 
reasonably) reported by the services as fully mission capable, actually 
have electronic combat systems with previously unknown faults. During 
our review, we found this to be true as a result of our direct observation of 
the new tester in use at Eglin. We observed four aircraft being tested for 
an upcoming Southern Watch deployment. In the testing that we observed, 
all four aircraft, which were believed to be fully mission capable, were 
found to have unknown faults that had to be repaired. 

The Air Force has a criterion that its F-15 fighter wings seek to maintain an 
81-percent fuUy mission capable rate. However, combining the statistics 
for using the new tester in 2000 at the Eglin wing (29 of 31 aircraft had 
unknown faults) and the Langley wing (12 of 13 had imknown faults), the 
Air Force found that 41 of 44 F-15Cs tested were not fully mission capable. 
Likewise, since all 10 of the Navy’s F/A-18C aircraft tested at Oceana Naval 
Air Station with the new tester had three or more unknown faults, the 
Navy also could face unacceptably low readiness levels. 


Once the services introduce the new tester for widespread usage, they are 
likely to find, as they did during testing, that the reliability of their 
electronic combat systems is much lower than previously thought. 
Consequently, more logistics support in the form of additional spare parts 
to fix previously undiagnosed faults will be required in the future. 

According to Air Force officials, on the basis of the new tester’s use on the 
F-16C aircraft at Eglin and Langley, the Air Force will experience a 
requirement for more frequent repairs and an added logistics problem. At 
Warner Robbins Air Logistics Center, we were advised that spare parts 
shortages already exist for F-15 electronic combat systems. Maintenance 
officials at both Eglin and Langley stated that these shortages cause them 
to use cannibalization—^i.e., removing a working part from one aircraft to 
install it on another aircraft—to meet the wing’s flying schedule. For 
example, while we observed the new tester being used on operational 
aircraft at Eglin, several cannibalizations of electronic combat system 
parts were required before the testing could be completed. Maintenance 
officials told us that because spare parts were in limited supply, it was 
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Maintenance Burden 


common for aircraft being tested to use cannibalized parts from another 
aircraft in order to be repaired.^ 

Although the scope of our review did not include an assessment of the 
impact of using the new tester on logistics for the Navy’s F/A-18C fleet, we 
believe that using the new tester could also reveal a significant future 
problem for F/A-18C operational deployments. Generally, even if the Navy 
does not have a spare parts shortage as serious as the Air Force’s, 
maintaining the readiness of deployed aircraft on carriers is more difficult 
because of the quantity limitations on spare parts storage aboard ship. A 
Navy maintenance person advised us that on his carrier’s recent 
deployment to Southern Watch, the spare parts for the electronic combat 
systems used on the F/A-18C were completely exhausted and maintenance 
personnel had to resort to cannibalization to maintain flight operations. 
This situation existed without the Navy’s having access to the new tester, 
which would likely identify even more parts needing to be replaced. 


Our review indicates that, in addition to the potential for heightened 
readiness and logistics concerns, the introduction of the new tester could 
increase the maintenance burden on the services because the new tester 
could identify many more repairs that have to be made. 'This could 
intensify existing pressures on maintenance personnel to resort to 
cannibalization. As we stated in our recent testimony for the Congress, 
making repairs via cannibalization requires at least twice the maintenance 
time as making repairs using new spare parts. Moreover, if use of the new 
tester results in further increases to the maintenance burden, it could also 
affect the Air Force’s problem in retaining skilled technicians. Reinforcing 
this, both Eglin and Langley maintenance officials advised us that there 
are already shortages of trained maintenance personnel at the 33rd and 1st 
wings. In fact, the Air Force Postm’e Statement 2000 cites low retention of 
maintenance technicians as one of four factors resulting in the 99-percent 
drop in the mission-capable rates of Air Force aircraft since 1994. 
Furthermore, given the test results associated with the use of the new 
tester on the F/A-18C, the Navy could expect a significant increase in its 
maintenance burden. However, we were not made aware of any particular 


® For an extensive discussion of cannibalization and its adverse effects, see Military 
Aircraft: Cannibalization Adversdy Affects Personnel and Maintenance fGAO-01-693T, 
May 22,2001). 
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retention problem associated with the maintenance burden being 
experienced by Navy personnel during this review. 


Reduced Electronic 
Combat Readiness Could 
Increase the Need for 
Suppression of Enemy Air 
Defenses 


Using New Tester on Other 
Aircraft Types Could 
Reveal Similar Problems 


The new tester’s use could cause pilots of Air Force and Navy combat 
aircraft to be reluctant to rely solely on their electronic combat systems 
for self-protection from enemy air defenses. Recognizing reduced 
readiness and reliability of their self-protection systems, pilots could look 
for greater support from other specialized aircraft designed to suppress 
enemy air defenses, such as the EA-6B. We recently reported that current 
suppression capabilities are not adequate.'* To the extent that the new 
tester discloses reliability problems with existing electronic combat self¬ 
protection systems, the need to improve suppression capabilities would 
only be that much greater. 


Given the experience from using the new tester on the F-15C and F/A-18C, 
it is likely that using the new tester will find a number of undisclosed 
faults in electronic combat systems. Many of the electronic combat 
systems on current aircraft are older systems that are already experiencing 
obsolescence problems, such as difficulty in acquiring spares due to 
vendors that go out of business or are no longer producing old technology 
equipment (referred to as “vanishing vendors”). The Air Force’s special 
test program, called Combat Shield, is used periodically to test a variety of 
types of operational aircraft for readiness. Typically, even without using 
the new tester, testing via Combat Shield has found that some aircraft in 
eveiy wing tested have faults in their electronic combat systems, 
regardless of the aircraft type. For example. Combat Shield found 
undisclosed faults when testing was conducted at wings equipped with the 
F-16. 


In fact. Air Force and Navy officials have already identified emei^g 
problems regarding readiness, logistics, and maintenance for other 
electronic combat systems. This applies to systems both internally carried 
or externally mounted on an aircraft. For example, the ALQ-131 jammer 
system, externally carried by several Air Force aircraft, is projected to 
have a mission capable rate of 30 to 40 percent by 2006 because of 
obsolescence and the lack of spares. Furthermore, according to Air Force 


* See Electronic Warfare: Comprehensive Strategy Needed for Suppressing Enemy Air 
Defenses (GAO-01-28, Jan. 3,2001). 
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officials at Warner Robbins Air Logistics Center, funding priorities have 
constrained both i^are parts acquisition and sustaining the engineering 
needed to address the obsolescent parts issue. 


Conclusion 


The armed services have had problems for years with their ability to 
adequately test their electronic combat systems. The success of the new 
tester in providing improved test capability is a positive development. 
Because the tester has identified many more faults in the F-15C and 
F/A-18C electronic combat systems than the current test equipment was 
identifying, existing readiness, logistics, and maintenance problems with 
such systems could worsen. However, pilots would at least have greater 
knowledge about the readiness and reliability of their self-protection 
systems and their need for support from specialized aircraft designed to 
suppress enemy air defenses. On balance, we believe it makes sense for 
the Air Force and Navy to consider using the new test equipment on their 
nonfighter aircraft. 


Recommendation for 
Executive Action 


Because the new tester’s use provides the ability to identify previously 
imknown faults in electronic combat systems, we recommend that the 
Secretary of Defense direct the Air Force and the Navy to consider 
expanding the new tester’s use beyond fighter aircraft to other types of 
aircraft. 


Agency Comments 


In written comments on a draft of this report, the Department of Defense 
agreed with our finding that the new tester provides a much better 
capability to assess electronic combat systems than the services’ existing 
testers. It also agreed that once the services introduce the new tester for 
use on a widespread basis, they are likely to find that the reliability of the 
electronic combat systems is lower than previously thought. 
Consequently, more logistics support may be reqxiired in the future, and 
the maintenance burden may increase. The Department concurred with 
our recommendation. 


Scope and 
Methodology 


We reviewed the results of the Joint Service Electronic Combat Systems 
Tester development testing and determined program status through 
discussions with program office officials and a review of appropriate 
documentation. We discussed the status of the Air Force’s aircraft 
electronic combat ^tems with Air Combat Command officials 
responsible for these systems on all Air Force operational aircraft. We held 
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discussions regarding logistics support and maintenance with officials at 
Warner Robbins Air Logistics Center responsible for Air Force electronic 
combat systems. We held similar discussions with officials at Jacksonville 
Naval Air Station regarding Navy aircraft electronic combat systems. We 
also observed and discussed the testing of operational F-15C aircraft with 
officials at the 33rd Wing at Eglin Air Force Base and discussed the results 
of similar tests with officials of the 1st Wing at Langley Air Force Base. 
These two Wings have about 40 percent of the Air Force’s F-15C aircraft. 
We also relied on our previous reviews of electronic warfare for 
backgroxmd information on the existing logistics and maintenance 
problems with electronic combat systems. 

We conducted our review from August 2000 to August 2001 in accordance 
with generally accepted government auditing standards. 


We are sending copies of this letter to the Secretaries of the Air Force and 
Navy; to interested congressional committees; and to the Director, Office 
of Management and Budget. If you have any questions, please contact me 
on (202) 512-4841. Mgjor contributors to this report were Michael Aiken, 
Terry Parker, and Charles Ward. 

Sincerely yours, 

R. E. Levin 

Director, Acquisition and 
Sourcing Management 
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Appendix I: Comments From the 
Department of Defense 


HE UNDER SECRETARY OF DEFENSE 

3000 DEFENSE PENTAGON 
WASHINGTON DC 20301-3000 

2 3 AUG ?001 

ACQUISrTION AND 
TECHNOLOGY 

Mr. R. E. Levin 

Director, Acquisition and Sourcing Management 
United States General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Levin: 

This is the Department of Defense (DoD) response to the General 
Accounting Office (GAO) draft report, “ELECTRONIC COMBAT: Services 
Should Consider Greater Use of New Test Equipment for Their Aircraft,'’ dated 
July 18, 2001(GAO Code 120068/OSD Case 01-843) 

Electronic combat (EC) operations are critically important in establishing 
and maintaining air supremacy. The Department agrees with the GAO’s 
conclusions regarding current shortcomings in EC test equipment. The new EC 
tester cited in the GAO report provides a much better capability to assess both 
internal and external aircraft EC systems than the Services’ existing test systems. 

The Department concurs with the GAO’s recommendation and appreciates 
the opportunity to respond to the draft report. 

Sincerely, 





George R. Schneiter 
Director 

Strategic and Tactical Systems 


Attachment 
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Appendix I: Comments From the Department 
of Defense 


GAO DRAFT REPORT, ‘^ELECTRONIC COMBAT: Services Should 
Consider Greater Use of New Test Equipment for Their Aircraft”, Dated 
July 18,2001 (Code 120068/Case 01-843) 


RECOMMENDATION ; Because use of the new tester provides the ability to identify 
previously unknown faults in electromc combat systems, we recommend that the Air 
Force and Navy consider expanding use of the new tester beyond fighter aircraft to other 
types of aircraft. 

POD RESPONSE ; Concur. The GAO based its reconomendation on the fact that the 
Services have had problems with the ability to adequately lest their electronic combat 
systems for years. The Department agrees that the new tester provides a much better 
capability to test aircraft electronic combat systems than the Services’ existing test 
systems. Furthermore, the Department agrees that once the Services introduce the new 
tester for use on a widespread basis, they are likely to find that the reliability of their 
electronic combat systems is lower than previously thought. Consequently, more 
logistics support in the form of additional spare parts to fix previously undiagnosed faults 
may be required in the future, and the maintenance burden on the Services may increase 
because the new tester will identify more repair needs. 


(120068) 
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